{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use PY in Linear Algebrea\n",
    "<br>\n",
    "\n",
    "## Chapter Zero 乘法和庫\n",
    "<br>\n",
    "### 矩陣和向量的乘法\n",
    "1. 點乘（dot product）：也被稱為「數量積」「內積」「點積」。結果的絕對值是一個向量在另一個向量方向上的投影的長度誠意另一個向量的長度，是一個標量。\n",
    "2. 叉乘（cross product）：也被稱為「向量積」「外積」。對於兩個 $n$ 緯的向量叉乘，結果是一個和 $n-1$ 個向量都垂直的向量，方向符合右手定則。\n",
    "3. 矩陣乘法：代表了線性變換的過程。\n",
    "<br>\n",
    "\n",
    "### python 的 sympy 库 和 numpy 库\n",
    "说明：sympy 用於模擬和理解，numpy 用於实践。\n",
    "<br> \n",
    "\n",
    "#### sympy\n",
    "`sympy`使用`*`代表矩陣乘法；<br>\n",
    "`sympy`使用`dot`表示點乘；<br>\n",
    "`sympy`使用`cross`表示叉乘<br>\n",
    "`det()`表示行列式；`inv()`表示擬矩陣；`adjugate()`伴隨矩陣<br>\n",
    "\n",
    "#### numpy\n",
    "`numpy`使用`*`表示`array`乘法默認是按照位置的乘法，矩陣`matrix`默認的是矩陣乘法，混合起來的話默認是矩陣乘法；<br>\n",
    "`numpy`使用`dot`表示矩陣`matrix`和`array`的矩陣乘法；<br>\n",
    "`numpy`使用`cross`表示叉乘；<br>\n",
    "`numpy`使用`det()`表示行列式；<br>\n",
    "`.T` 返回自身的轉置；<br>\n",
    "`.H` 返回自身的共軛轉置；<br>\n",
    "`.J` 返回自身的逆矩陣；<br>\n",
    "`.A` 返回自身數據的2維數組的一個視圖（沒有做任何的拷貝）<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chapter One Matrix \n",
    "\n",
    "數學上，一個$m\\times n$的矩陣是由一個$m$行(row)$n$列(column)元素排列成的矩形陣列，矩陣裡的元素可以是數字、符號或者數學式。在圖像處理、人工智能燈領域，使用矩陣來表示和處理數據非常常見。例：<br>\n",
    "$\\mathbf{A}_{2 \\times 3}=\\begin{bmatrix} 5 & 2 & 7 \\\\ 1 & 3 & 4  \\end{bmatrix}$\n",
    "<br>\n",
    "其中，矩陣$A$的下標$2\\times 3$表示$A$是一個2行3列的矩陣。類似的，另一個示例：<br>\n",
    "$\\mathbf{ B }_{ 4 \\times 4 }=\\begin{bmatrix} 5 & 2 & 7 & 6 \\\\ 1 & 3 & 4 & 2 \\\\ 7 & -1 & 9 & 0 \\\\ 8  & 2 & -2 & 3 \\end{bmatrix}$<br>\n",
    "特別的，如果我們要表示矩陣$A$的第二行第二個元素：3，可以使用$A[2,2]$或者$a_{2,2}$。<br><br>\n",
    "在`python`中的`numpy`庫中提供了`ndarrary`類用於儲存高維數組及普通的數組運算，另外提供了`matrix`類用來支持矩陣運算（矩陣乘法）。下面來舉個例子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[5 2 7]\n",
      " [1 3 4]]\n",
      "[[15  2  7  6]\n",
      " [ 1  3  4  2]\n",
      " [ 7 -1  9  0]\n",
      " [ 8  2 -2  3]]\n"
     ]
    }
   ],
   "source": [
    "#first method to store the matrix\n",
    "import numpy as np\n",
    "\n",
    "a = np.matrix('5 2 7;1 3 4')\n",
    "b = np.matrix('15 2 7 6;1 3 4 2;7 -1 9 0;8 2 -2 3')\n",
    "print(a)#print matrix a\n",
    "print(b)#print matrix b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[5 2 7]\n",
      " [1 3 4]]\n",
      "[[ 5  2  7  6]\n",
      " [ 1  3  4  2]\n",
      " [ 8  2 -2  3]]\n"
     ]
    }
   ],
   "source": [
    "#second method to store the matrix\n",
    "import numpy as np\n",
    "\n",
    "a = np.matrix([[5,2,7],[1,3,4]])\n",
    "b = np.matrix([[5,2,7,6],[1,3,4,2],[8,2,-2,3]])\n",
    "print(a)#print matrix a\n",
    "print(b)#print matrix b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這兩種形式完全等效，但是第一種更加直觀，不容易犯錯，所以推薦第一種方式（也很類似`LaTeX`中構成矩陣的寫法）；<br>\n",
    "要把一個`matrix`的對象轉換為`ndarray`對象，可以直接用`getA()`的方法。如果吧`ndarray`對象轉換成`matrx`對象可以用`asmatrix()`方法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[5 2 7]\n",
      " [1 3 4]]\n",
      "<class 'numpy.ndarray'>\n",
      "[[5 2 7]\n",
      " [1 3 4]]\n",
      "<class 'numpy.matrixlib.defmatrix.matrix'>\n"
     ]
    }
   ],
   "source": [
    "b = a.getA()#turn a into ndarray\n",
    "print(b)\n",
    "print(type(b))# the type of b\n",
    "c = np.asmatrix(b)\n",
    "print(c)\n",
    "print(type(c))# the type of c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果你要取出矩陣中的某個值，可以使用類似於數組的下標運算符。但是要注意，計算機的計數方式是從0開始的。例如，要取出$A[2,2]$，應該使用："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[1,1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 運算\n",
    "\n",
    "#### 加\n",
    "矩陣加法的定義非常符合直覺。假設有$\\mathbf{ A }_{ 3 \\times 3 }=\\begin{bmatrix} 1 & 0 & 1 \\\\ 1 & 2 & 1 \\\\ 2 & 1 & 1 \\end{bmatrix}$，$\\mathbf{ B }_{ 3 \\times 3 }=\\begin{bmatrix} 2 & 1 & -1 \\\\ 0 & -1 & 2 \\\\ 2 & -1 & 0 \\end{bmatrix}$，<br> 則：\n",
    "$\\mathbf{A}+\\mathbf{B} = \\begin{bmatrix} 1 & 0 & 1 \\\\ 1 & 2 & 1 \\\\ 2 & 1 & 1 \\end{bmatrix} + \\begin{bmatrix} 2 & 1 & -1 \\\\ 0 & -1 & 2 \\\\ 2 & -1 & 0 \\end{bmatrix} = \\begin{bmatrix} 1+2 & 0+1 & 1+(-1) \\\\ 1+ 0 & 2+(-1) & 1+2 \\\\ 2+2 & 1+(-1) & 1+0 \\end{bmatrix} = \\begin{bmatrix} 3 & 1 & 0 \\\\ 1 & 1 & 3 \\\\ 4 & 0 & 1 \\end{bmatrix}$\n",
    "<br>\n",
    "要注意，兩個矩陣的行數和列數必須相同，否則無定義。下面是`python`示例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[3 1 0]\n",
      " [1 1 3]\n",
      " [4 0 1]]\n"
     ]
    }
   ],
   "source": [
    "a = np.matrix('1 0 1;1 2 1;2 1 1')\n",
    "b = np.matrix('2 1 -1;0 -1 2;2 -1 0')\n",
    "print(a+b)# print a+b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "很容易看出，矩陣的加法滿足交換律和結合律，即$A+B=B+A$，$(A+B)+C=A+(B+C)$。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 減\n",
    "矩陣減法也和加法一樣簡單。對於上面給出的$A$和$B$，有：<br>\n",
    "$\\mathbf{A}-\\mathbf{B}=\\begin{bmatrix} 1 & 0 & 1 \\\\ 1 & 2 & 1 \\\\ 2 & 1 & 1 \\end{bmatrix}-\\begin{bmatrix} 2 & 1 & -1 \\\\ 0 & -1 & 2 \\\\ 2 & -1 & 0 \\end{bmatrix}=\\begin{bmatrix} 1-2 & 0-1 & 1-(-1) \\\\ 1-0 & 2-(-1) & 1-2 \\\\ 2-2 & 1-(-1) & 1-0 \\end{bmatrix}=\\begin{bmatrix} -1 & -1 & 2 \\\\ 1 & 3 & -1 \\\\ 0 & 2 & 1 \\end{bmatrix}$<br>\n",
    "同樣，相減的兩個矩陣行數和列數必須完全相同，否則無定義。下面是`python`示例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-1 -1  2]\n",
      " [ 1  3 -1]\n",
      " [ 0  2  1]]\n"
     ]
    }
   ],
   "source": [
    "print(a-b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 乘\n",
    "矩陣乘法的定義是$A_{i\\times j}$矩陣的每一行的元素分別與$B_{j\\times k}$矩陣的每一列的元素兩兩相乘並詳加，從而，得到新的矩陣$C_{j\\times k$。兩個矩陣能相乘的充份必要條件是第一個矩陣的列數等於第二個矩陣的行數相等，否則，無定義。$\\begin {aligned} \n",
    "\\mathbf{A} \\times \\mathbf{B} &=\\begin{bmatrix} 1 & 0 & 1 \\\\ 1 & 2 & 1 \\\\ 2 & 1 & 1 \\end{bmatrix}\\times \\begin{bmatrix} 2 & 1 & -1 \\\\ 0 & -1 & 2 \\\\ 2 & -1 & 0 \\end{bmatrix} \\\\\\ &=\\begin{bmatrix} 1\\cdot 2+0\\cdot 0+1\\cdot 2 & 1\\cdot 1+0\\cdot (-1)+1\\cdot (-1) & 1\\cdot (-1)+0\\cdot 2+1\\cdot 0 \\\\ 1\\cdot 2+2\\cdot 0+1\\cdot 2 & 1\\cdot 1+2\\cdot (-1)+1\\cdot (-1) & 1\\cdot (-1)+2\\cdot 2+1\\cdot 0 \\\\ 2\\cdot 2+1\\cdot 0+1\\cdot 2 & 2\\cdot 1+1\\cdot (-1)+1\\cdot (-1) & 2\\cdot (-1)+1\\cdot 2+1\\cdot 0 \\end{bmatrix}\\\\\\ &=\\begin{bmatrix} 4 & 0 & -1 \\\\ 4 & -2 & 3 \\\\ 6 & 0 & 0 \\end{bmatrix}\n",
    "\\end {aligned}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "再舉一個行列數不同的例子，假設有$\\mathbf{C}_{2 \\times 3} = \\begin{bmatrix} 5 & 7 & 2 \\\\ 4 & 3 & 1 \\end{bmatrix}$和$\\mathbf{D}_{3 \\times 1} = \\begin{bmatrix} 1 \\\\ 5 \\\\ 6 \\end{bmatrix}$，那麼，我們可以得出：$\\mathbf{C}\\times \\mathbf{D} = \\begin{bmatrix} 5 & 7 & 2 \\\\ 4 & 3 & 1 \\end{bmatrix}\\times \\begin{bmatrix} 1 \\\\ 5 \\\\ 6 \\end{bmatrix}\n",
    " =\\begin{bmatrix} 5 \\cdot 1+ 7 \\cdot 5+ 2\\cdot 6 \\\\ 4\\cdot 1+3\\cdot 5+1\\cdot 6  \\end{bmatrix} =\\begin{bmatrix} 52 \\\\ 25 \\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "與初等代數中學的乘法不同，矩陣的乘法並不滿足交換律，即，$A \\times B \\neq B \\times A$，但是，它滿足分配律：$(A\\times B)\\times C = A\\times (B\\times C)$。<br>\n",
    "再介紹兩個特殊的矩陣：<br>\n",
    "1. 單位矩陣（Identity Matrix）他的特點是與行數和列數相等，對角線上的值都是1，其他值都是0。另一個特點是：它與任意矩陣$A$相乘結果等於$A$。<br>\n",
    "$\\mathbf{I}_{3 \\times 3} = \\begin{bmatrix} 1 & 0 & 0 \\\\ 0 & 1 & 0 \\\\ 0 & 0 & 1 \\end{bmatrix}$這是一個 $3\\times 3$的單位矩陣<br>\n",
    "<br>\n",
    "2. 零矩陣（Null Matrix）顧名思義就是全部元素都是0的矩陣。零矩陣乘以任何矩陣都是零矩陣，與任何矩陣$A$都等於$A$。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "this is a times b:\n",
      "[[ 4  0 -1]\n",
      " [ 4 -2  3]\n",
      " [ 6  0  0]]\n",
      "this is b times a:\n",
      "[[ 1  1  2]\n",
      " [ 3  0  1]\n",
      " [ 1 -2  1]]\n"
     ]
    }
   ],
   "source": [
    "#python示例\n",
    "print(\"this is a times b:\")\n",
    "print(a*b)\n",
    "print(\"this is b times a:\")\n",
    "print(b*a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "this is c times d:\n",
      "[[52]\n",
      " [25]]\n",
      "this is a times b times d:\n",
      "[[-2]\n",
      " [12]\n",
      " [ 6]]\n",
      "first,we user b times d, then we use a to times the b*d:\n",
      "[[-2]\n",
      " [12]\n",
      " [ 6]]\n",
      "This is a times I:\n",
      "[[1. 0. 1.]\n",
      " [1. 2. 1.]\n",
      " [2. 1. 1.]]\n",
      "This is I times a:\n",
      "[[1. 0. 1.]\n",
      " [1. 2. 1.]\n",
      " [2. 1. 1.]]\n",
      "This is a times z:\n",
      "[[0. 0. 0.]\n",
      " [0. 0. 0.]\n",
      " [0. 0. 0.]]\n",
      "This is z times a:\n",
      "[[0. 0. 0.]\n",
      " [0. 0. 0.]\n",
      " [0. 0. 0.]]\n"
     ]
    }
   ],
   "source": [
    "c = np.matrix('5 7 2;4 3 1')\n",
    "d = np.matrix('1;5;6')\n",
    "print(\"this is c times d:\")\n",
    "print(c*d)\n",
    "print(\"this is a times b times d:\")\n",
    "print(a*b*d)\n",
    "print(\"first,we user b times d, then we use a to times the b*d:\")\n",
    "print(a*(b*d))\n",
    "I = np.eye(3)\n",
    "#we defined I as a 3 by 3 indentity matrix;eye() equals to'I = np.matrix('1 0 0;0 1 0;0 0 1')'\n",
    "print(\"This is a times I:\")\n",
    "print(a*I)\n",
    "print(\"This is I times a:\")\n",
    "print(I*a)\n",
    "z = np.zeros([3,3])\n",
    "#we define z as a 3 by 3 null matrix; np.zeros() equals to'z = np.matrix('0 0 0;0 0 0;0 0 0')'\n",
    "print(\"This is a times z:\")\n",
    "print(a*z)\n",
    "print(\"This is z times a:\")\n",
    "print(z*a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ~~除~~（求逆）\n",
    "<br>\n",
    "矩陣中沒有除法這種操作，但是有一個相似的運算，叫做求逆。在線性代數中，給定一個$n$階的方陣$A$ ，若存在一個$n$階矩陣$B$，使得$AB = BA = I_n$，其中$I_n$為$n$階的單位矩陣，則稱$A$是**可逆**的，而$B$則是$A$的逆矩陣，記作：$A^{-1}$。<br>\n",
    "##### 伴隨陣法：\n",
    "$\\mathbf{A}^{-1}=\\frac{1}{|\\mathbf{A}|}\\begin{bmatrix}A_{11} & A_{21} & \\ldots & A_{n1} \\\\ A_{12} & A_{22} & \\ldots & A_{n2} \\\\ \\ldots & \\ldots & \\ldots & \\ldots \\\\ A_{1n} & A_{2n} & \\ldots & A_{nn} \\end{bmatrix}$，其中$A_{ij}$是$|A|$中元素$a_{ij}$的代數余子式。\n",
    "<br>\n",
    "矩陣$\\begin{bmatrix}A_{11} & A_{21} & \\ldots & A_{n1} \\\\ A_{12} & A_{22} & \\ldots & A_{n2} \\\\ \\ldots & \\ldots & \\ldots & \\ldots \\\\ A_{1n} & A_{2n} & \\ldots & A_{nn} \\end{bmatrix}$稱為矩陣$A$ 的伴隨矩陣，記作$A*$\n",
    "<br>\n",
    "##### 初等變換法：\n",
    "如果矩陣$A$和$B$互逆，則$AB=BA=I$。由$AB=BA$可以得出：兩個矩陣都是方陣；由$AB=I$以及定理“兩個矩陣的乘積的行列式等於兩個矩陣的行列式的乘積”可得：兩個矩陣的行列式不等於零。現在假設三階矩陣${ \\mathbf{A} }_{ 3 \\times 3 }=\\begin{bmatrix} 1 & 0 & 1 \\\\ 1 & 2 & 1 \\\\ 2 & 1 & 1 \\end{bmatrix}$，則：<br>\n",
    "$\\begin{aligned}\n",
    "\\begin{bmatrix}\\mathbf{A} \\mathbf{I}\\end{bmatrix} \n",
    "& \\rightarrow \n",
    "\\begin{bmatrix} \n",
    "1 & 0 & 1 & 1 & 0 & 0 \\\\\n",
    "1 & 2 & 1 & 0 & 1 & 0 \\\\\n",
    "2 & 1 & 1 & 0 & 0 & 1\n",
    "\\end{bmatrix} \n",
    "\\rightarrow \n",
    "\\begin{bmatrix} \n",
    "1 & 0 & 1 & 1 & 0 & 0 \\\\\n",
    "0 & 2 & 0 & -1 & 1 & 0 \\\\\n",
    "2 & 1 & 1 & 0 & 0 & 1\n",
    "\\end{bmatrix}\n",
    "\\rightarrow \n",
    "\\begin{bmatrix} \n",
    "1 & 0 & 1 & 1 & 0 & 0 \\\\\n",
    "0 & 1 & 0 & -0.5 & 0.5 & 0 \\\\\n",
    "2 & 1 & 1 & 0 & 0 & 1\n",
    "\\end{bmatrix}\\\\\n",
    "& \\rightarrow \n",
    "\\begin{bmatrix} \n",
    "1 & 0 & 1 & 1 & 0 & 0 \\\\\n",
    "0 & 1 & 0 & -0.5 & 0.5 & 0 \\\\\n",
    "1 & 1 & 0 & -1 & 0 & 1\n",
    "\\end{bmatrix}\n",
    "\\rightarrow \n",
    "\\begin{bmatrix} \n",
    "1 & 0 & 1 & 1 & 0 & 0 \\\\\n",
    "0 & 1 & 0 & -0.5 & 0.5 & 0 \\\\\n",
    "1 & 0 & 0 & -0.5 & -0.5 & 1\n",
    "\\end{bmatrix}\n",
    "\\rightarrow \n",
    "\\begin{bmatrix} \n",
    "0 & 0 & 1 & 1.5 & 0.5 & -1 \\\\\n",
    "0 & 1 & 0 & 0 & 0.5 & 0 \\\\\n",
    "1 & 0 & 0 & -0.5 & -0.5 & 1\n",
    "\\end{bmatrix}\\\\\n",
    "&\\rightarrow \n",
    "\\begin{bmatrix} \n",
    "1 & 0 & 0 & -0.5 & -0.5 & 1 \\\\\n",
    "0 & 1 & 0 & -0.5 & 0.5 & 0 \\\\\n",
    "0 & 0 & 1 & 1.5 & 0.5 & -1\n",
    "\\end{bmatrix}\n",
    "\\end{aligned}$\n",
    "<br>\n",
    "所以：$\\mathbf{A}^{-1}=\\begin{bmatrix}-0.5 & -0.5 & 1 \\\\ -0.5 & 0.5 & 0 \\\\ 1.5 & 0.5 & -1\\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 奇異矩陣（Singular Matrix）\n",
    "如果一個方陣$A$滿足條件$|A|(det(A))\\neq 0$，則稱$A$為非奇異矩陣（non-singular matrix），否則則稱為奇異矩陣。簡單的說**沒有逆矩陣的矩陣就是奇異矩陣，有逆矩陣的矩陣就是非奇異矩陣**<br>\n",
    "`python`求逆示例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-5.00000000e-01 -5.00000000e-01  1.00000000e+00]\n",
      " [-5.00000000e-01  5.00000000e-01 -1.85037171e-17]\n",
      " [ 1.50000000e+00  5.00000000e-01 -1.00000000e+00]]\n",
      "[[1. 0. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 0. 1.]]\n"
     ]
    },
    {
     "ename": "LinAlgError",
     "evalue": "Singular matrix",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mLinAlgError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-9-1bb88e56f1ce>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mI\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m#a times a inverse equals to indentity matrix\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmatrix\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'0 1;0 0'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mI\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;31m#because f has no inverse, f is a singular matrix\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/matrixlib/defmatrix.py\u001b[0m in \u001b[0;36mgetI\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    923\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    924\u001b[0m             \u001b[0;32mfrom\u001b[0m \u001b[0mnumpy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdual\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mpinv\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 925\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0masmatrix\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    926\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    927\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mgetA\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/linalg/linalg.py\u001b[0m in \u001b[0;36minv\u001b[0;34m(a)\u001b[0m\n\u001b[1;32m    526\u001b[0m     \u001b[0msignature\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'D->D'\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misComplexType\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;34m'd->d'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    527\u001b[0m     \u001b[0mextobj\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_linalg_error_extobj\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_raise_linalgerror_singular\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 528\u001b[0;31m     \u001b[0mainv\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_umath_linalg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msignature\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msignature\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mextobj\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mextobj\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    529\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mwrap\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mainv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mastype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult_t\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    530\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/linalg/linalg.py\u001b[0m in \u001b[0;36m_raise_linalgerror_singular\u001b[0;34m(err, flag)\u001b[0m\n\u001b[1;32m     87\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     88\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_raise_linalgerror_singular\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mflag\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 89\u001b[0;31m     \u001b[0;32mraise\u001b[0m \u001b[0mLinAlgError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Singular matrix\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     90\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     91\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_raise_linalgerror_nonposdef\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mflag\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mLinAlgError\u001b[0m: Singular matrix"
     ]
    }
   ],
   "source": [
    "a = np.matrix('1 0 1; 1 2 1; 2 1 1')\n",
    "print(a.I) #the inverse of matrix a \n",
    "print(a*a.I) #a times a inverse equals to indentity matrix\n",
    "f = np.matrix('0 1;0 0')\n",
    "print(f.I)#because f has no inverse, f is a singular matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 矩陣的轉置（Transport）\n",
    "在線性代數中，矩陣$A$的轉置是另一個矩陣$A^T$，如下所示：<br>\n",
    "* 把$A$的橫行寫成$A^T$的縱列；\n",
    "* 把$A$的縱列寫成$A^T$的橫行；\n",
    "形式上說就是，$m\\times n$的矩陣$A$的轉置是$n\\times m$矩陣。<br>\n",
    "矩陣$\\underset{m\\times n}{\\mathbf{A}} = \\begin{bmatrix}a_{11} & a_{12} & \\ldots & a_{1n} \\\\ a_{21} & a_{22} & \\ldots & a_{2n} \\\\ \\ldots \\\\ a_{m1} & a_{m2} & \\ldots & a_{mn}\\end{bmatrix}$的轉置定義為$\\underset{n\\times n}{A^{T}} = \\begin{bmatrix}a_{11} & a_{21} & \\ldots & a_{m1} \\\\ a_{12} & a_{22} & \\ldots & a_{m2} \\\\ \\ldots \\\\ a_{1n} & a_{2n} & \\ldots & a_{mn}\\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#python 示例：\n",
    "a = np.matrix('2 4;1 3')\n",
    "print(a.T)#the transport of a\n",
    "b = np.matrix('1 2 3;4 5 6')\n",
    "print(b.T)#the transport of b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**特別的，矩陣的轉置有一個性質：矩陣的轉置等於矩陣調換後分別做轉置的乘積：$（A\\bullet B)^T = B^T\\bullet A^T$**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.matrix('2 4;1 3')\n",
    "b = np.matrix('1 6;2 5')\n",
    "c = a*b\n",
    "print(c.T)\n",
    "print(b.T*a.T)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from pylab import *\n",
    "\n",
    "xlim(-3, 3)\n",
    "ylim(0, 3.5)\n",
    "plt.plot([0,2.5],[3.5,0],'blue',[-1,3],[0,3.5],'red')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以用矩陣表示成：$\\begin{bmatrix}\n",
    "3 & 2 \\\\\n",
    "-1 & 1\n",
    "\\end{bmatrix}\n",
    "\\begin{bmatrix}\n",
    "x \\\\\n",
    "y\n",
    "\\end{bmatrix}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "7\\\\\n",
    "1\n",
    "\\end{bmatrix}$\n",
    "<br>\n",
    "設$\\begin{bmatrix}3 & 2 \\\\-1 & 1\\end{bmatrix}$為矩陣$A$，將等式兩邊左乘一個$A$的逆，就可以得到：<br>\n",
    "$\\begin{aligned}\n",
    "A^{-1}A\n",
    "\\begin{bmatrix}\n",
    "x \\\\ y\n",
    "\\end{bmatrix}\n",
    "&= \n",
    "A^{-1}\n",
    "\\begin{bmatrix}\n",
    "7\\\\\n",
    "1\n",
    "\\end{bmatrix}\\\\\\\n",
    "&=\n",
    "\\frac{1}{|A|}\\begin{bmatrix}1 & -2 \\\\ 1 & 3\\end{bmatrix}\n",
    "\\begin{bmatrix}\n",
    "7\\\\\n",
    "1\n",
    "\\end{bmatrix}\\\\\\\n",
    "&=\n",
    "\\frac{1}{5}\\begin{bmatrix}1 & -2 \\\\ 1 & 3\\end{bmatrix}\n",
    "\\begin{bmatrix}\n",
    "7\\\\\n",
    "1\n",
    "\\end{bmatrix}\\\\\\\n",
    "&=\n",
    "\\frac{1}{5}\\begin{bmatrix}5 \\\\ 10\\end{bmatrix}\n",
    "\\end{aligned}$<br>\n",
    "因此，$\\begin{bmatrix}x \\\\ y\\end{bmatrix}=\\begin{bmatrix}1 \\\\ 2\\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "雖然這個方法在二元一次方程這看起來還沒有初中學的校園法解得快，但使用矩陣的好處在於對於更高維的數據，比如成白上千的未知數，這個解法依然有效。<br>\n",
    "在`python`中，我們可以使用`numpy`的線性代數算法庫`linalg`提供的`solve`方法來求解方程組。例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.matrix('3 2; -1 1')\n",
    "b = np.matrix('7; 1')\n",
    "print(np.linalg.solve(a, b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 求向量組合\n",
    "假設有向量$\\vec { a }  =  \\begin{bmatrix} 3 \\\\ -1 \\end{bmatrix}$，$\\vec { b }  =  \\begin{bmatrix} 2 \\\\ 1 \\end{bmatrix}$，求兩者如何組成新向量$\\vec { c }  =  \\begin{bmatrix} 7 \\\\ 1 \\end{bmatrix}$？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "soa = np.array([[0, 0, 3, -1], [0, 0, 2, 1], [0, 0, 7, 1]])\n",
    "X, Y, U, V = zip(*soa)\n",
    "plt.figure()\n",
    "ax = plt.gca()\n",
    "ax.quiver(X, Y, U, V, angles='xy', scale_units='xy', scale=1)\n",
    "ax.set_xlim([-1, 10])\n",
    "ax.set_ylim([-1, 10])\n",
    "plt.draw()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果用$x$和$y$分別表示兩個向量的背書，這個問題就可以用矩陣表示成：$\\begin{bmatrix}3 \\\\-1 \\end{bmatrix}x + \\begin{bmatrix}2 \\\\1\\end{bmatrix}y=\\begin{bmatrix}7\\\\1\\end{bmatrix}$<br>\n",
    "這樣就把這個問題轉換成和上一個問題同構了，使用同樣的方法可以得出：$\\begin{bmatrix}x \\\\ y\\end{bmatrix}=\\begin{bmatrix}1 \\\\ 2\\end{bmatrix}$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "soa = np.array([[0, 0, 3, -1], [0, 0, 4, 2], [0, 0, 7, 1],[4,2,3,-1]])\n",
    "X, Y, U, V = zip(*soa)\n",
    "plt.figure()\n",
    "ax = plt.gca()\n",
    "ax.quiver(X, Y, U, V, angles='xy', scale_units='xy', scale=1)\n",
    "ax.set_xlim([-1, 10])\n",
    "ax.set_ylim([-1, 10])\n",
    "plt.draw()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chapter Two  Determinant\n",
    "\n",
    "### 什麼是行列式\n",
    "\n",
    "**行列式**（Dterminant），在顯示代數中，行列式是一個值，一個數值可以告訴我們一個矩陣是否是奇異的。對於每一個$n\\times n$的矩陣$A$，據對應一個標量$det(A)$，它的值將告訴我們矩陣是否為非奇異的。\n",
    "<br>\n",
    "1. $2\\times 2$ 矩陣的行列式值\n",
    "<br>\n",
    "$det(\\   \n",
    "\\left[\\begin{array}{cc}   \n",
    "    a & b\\\\   \n",
    "    c & d\\\\      \n",
    "\\end{array}\\right]   \n",
    "\\  ) = ad - bc$<br>\n",
    "我們也可以用`sympy`進行核對："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sympy import *\n",
    "   \n",
    "a,b,c,d = symbols(\"a,b,c,d\")\n",
    "M1=Matrix([[a, b], [c, d]])\n",
    "print(M1.det())# print the determinant of matrix M1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 行列式的幾何意義：二維和三維歐式空間中的例子\n",
    "\n",
    "行列式的一個自然的怨氣就是$n$維平行體的體積。行列式的定義和$n$維平行體的體積有著本質上的關聯。<br>\n",
    "在二維空間中，行列式如上面例子所示；比如說：$det(A,B) =\\   \n",
    "\\left|\\begin{array}{cc}   \n",
    "    2 & 3\\\\   \n",
    "    1 & 4\\\\      \n",
    "\\end{array}\\right|   \n",
    "\\ = 2\\bullet 4 - 3\\bullet 1 = 5$，我們經過計算可以知道，當係數是實數的時候，行列式表示的就是向量$\\vec A$和$\\vec B$ 形成的平行四邊形的**有向面積**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from pylab import *\n",
    "\n",
    "plt.plot([0,2],[0,1],'blue',[3,5],[4,5],'blue',[0,3],[0,4],'red',[2,5],[1,5],'red')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "   \n",
    "a = np.matrix([[2,1],[3,4]])\n",
    "print(np.linalg.det(a))# print the determinant of matrix M1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們通過計算，可以得到這個矩陣的行列式等於$5$，可以看出，上圖由紅線和藍線形成的是一個平行四邊形，我們可以通過計算得出，它的面積等於$5$<br>\n",
    "*這裡只是簡單介紹二維行列式，具體請查閱課本**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "那如果是三位空間呢？在三維的有向空間中，三個三維向量的行列式是$\\left|\\begin{array}{ccc}   \n",
    "    x & x' & x''\\\\   \n",
    "    y & y' & z''\\\\\n",
    "    z & z' & z''\n",
    "\\end{array}\\right|  \n",
    "\\ = xy'z'' + x'y''z + x''yz' - xy''z' - xyz'' - x''y'z $\n",
    "<br>\n",
    "比如說，三個向量$(2,1,5),(6,0,8)$ 和 $(3,2,4)$ 的行列式為：\n",
    "$\\left|\\begin{array}{ccc}   \n",
    "    2 & 6 & 3\\\\   \n",
    "    1 & 0 & 2\\\\\n",
    "    5 & 8 & 4\n",
    "\\end{array}\\right| = 28\n",
    "$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "\n",
    "p0 = [2, 1,5]\n",
    "p1 = [6,0,8]\n",
    "p2 = [3,2,4]\n",
    "\n",
    "origin = [0,0,0]\n",
    "X, Y, Z = zip([0,0,0],origin,origin) \n",
    "U, V, W = zip(p0,p1,p2)\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111, projection='3d')\n",
    "ax.quiver(X,Y,Z,U,V,W,arrow_length_ratio=0.001)\n",
    "ax.set_xlim([-1, 5])\n",
    "ax.set_ylim([-1, 5])\n",
    "ax.set_zlim([-1, 8])\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "類比一下二維向量的行列式，我們可以猜測一下三維向量組的行列式的幾何意義，就是這三個向量行程的平行六面體的**有向體積**<br>；我們可以用`python`來算它的行列式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.matrix([[2,1,5],[6,0,8],[3,2,4]])\n",
    "print(np.linalg.det(a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***特別的，如果向量組中，所有向量共線或者共面（線性相關），面積或體積為零**，什麼是線性相關，我們會在後面解釋*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chapter Three Vector\n",
    "\n",
    "### What is Vector\n",
    "\n",
    "在初中（或高中）物理，我們就已經接觸過向量了，那向量到底是什麼？它是指一個同事具有大小和方向，且滿足平行四邊形法則的集合對象。一般地，同時滿足具有大小和方向兩個性質的集合對象就可以稱作為向量（物理中的電流：有方向，有大小，但是它不是向量，這是特殊情況，我們在線性代數中不討論）。與向量相對的叫做**標量**（scalar），一般標量和向量最大的區別就是是否有向。<br>\n",
    "在$R^n$空間中定義$\\vec V$，可以用一個包含$n$個實數的有序集來表示：$\\vec V = \\begin{bmatrix} v_1 \\\\ v_2 \\\\ \\dots \\\\ v_n\\end{bmatrix}$，這個有序集中的每一個元素被稱為向量的分量（在某一維度的量）。例如，一個在$R^2$空間的向量$\\begin{bmatrix} 2 \\\\ 1 \\end{bmatrix}$,我們也可以用$(2,1)$或$<2,1>$來表示："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "soa = np.array([[0, 0, 2, 1]])\n",
    "X, Y, U, V = zip(*soa)\n",
    "plt.figure()\n",
    "ax = plt.gca()\n",
    "ax.quiver(X, Y, U, V, angles='xy', scale_units='xy', scale=1)\n",
    "ax.set_xlim([0, 5])\n",
    "ax.set_ylim([0, 5])\n",
    "plt.draw()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "向量的長度被定義為$\\parallel v \\parallel = \\sqrt{v_1^2 + v_2^2 +\\dots + v_n^2}$；當然的，當$\\parallel v \\parallel = 1$，我們稱這個向量是**單位向量**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 基本運算\n",
    "\n",
    "#### 加法\n",
    "\n",
    "向量$a$和向量$b$的加法定義為：$\\mathbf{a} + \\mathbf{b} = \\begin{bmatrix} a_1 + b_2 \\\\ a_2 + b_2 \\\\ \\dots \\\\a_n + b_n \\end{bmatrix}$<br>\n",
    "下面通過畫圖來示意向量$\\mathbf{a} = \\begin{bmatrix} -1 \\\\ 2 \\end{bmatrix}$與$\\mathbf{b} = \\begin{bmatrix}3 \\\\ 1\\end{bmatrix}$相加，等於$\\begin{bmatrix} 2 \\\\ 3 \\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "soa = np.array([[0, 0, -1, 2], [0, 0, 3, 1], [0, 0, 2, 3],[3,1,-1,2]])\n",
    "X, Y, U, V = zip(*soa)\n",
    "plt.figure()\n",
    "ax = plt.gca()\n",
    "ax.quiver(X, Y, U, V, angles='xy', scale_units='xy', scale=1)\n",
    "ax.set_xlim([-2, 5])\n",
    "ax.set_ylim([-1, 5])\n",
    "plt.draw()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看出，向量的加法是遵從平行四邊形法則的，在`python`中，我們可以直接用`numpy`的`ndarray`來表示向量："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "a = np.array([-1,2])\n",
    "b = np.array([3,1])\n",
    "print(a+b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 減法\n",
    "\n",
    "了解了加法了，減法就相當於加了一個反方向的向量，$\\mathbf{a} - \\mathbf{b} = \\begin{bmatrix} a_1 - b_1 \\\\ a_2 - b_2 \\\\ \\ldots \\\\ a_n - b_n \\end{bmatrix}$，下面通過畫圖來示意向量$\\mathbf{a} = \\begin{bmatrix} -1 \\\\ 2 \\end{bmatrix}$與$\\mathbf{b} = \\begin{bmatrix}3 \\\\ 1\\end{bmatrix}$相減，等於$\\begin{bmatrix} -4 \\\\ 1 \\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "soa = np.array([[0, 0, -1, 2], [0, 0, -3, -1], [0, 0, -4, 1],[-3,-1,-1,2]])\n",
    "X, Y, U, V = zip(*soa)\n",
    "plt.figure()\n",
    "ax = plt.gca()\n",
    "ax.quiver(X, Y, U, V, angles='xy', scale_units='xy', scale=1)\n",
    "ax.set_xlim([-5, 3])\n",
    "ax.set_ylim([-3, 3])\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "a = np.array([-1, 2])\n",
    "b = np.array([3, 1])\n",
    "print(a - b)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 乘法\n",
    "\n",
    "##### 標量乘以向量\n",
    "\n",
    "標量$c$乘以向量$\\mathbf{a}$定義為：$c \\cdot \\mathbf{a} = \\begin{bmatrix} c \\cdot a_1 \\\\ c \\cdot a_2 \\\\ \\ldots \\\\ c \\cdot a_n \\end{bmatrix} = \\begin{bmatrix} a_1 \\cdot c \\\\ a_2 \\cdot c \\\\ \\ldots \\\\ a_n \\cdot c \\end{bmatrix}$<br>\n",
    "用圖來示意$\\mathbf{a} \\begin{bmatrix} -1 \\\\2 \\end{bmatrix}$乘以一個標量$3$得到$\\begin{bmatrix}-3 \\\\ 6 \\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "a = np.array([-1,2])\n",
    "print(3 * a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 向量點積\n",
    "\n",
    "定義如下：$\\vec{\\mathbf{a}}\\cdot \\vec{\\mathbf{b}} = \\begin{bmatrix} a_1 \\\\ a_2 \\\\ \\ldots \\\\ a_n\\end{bmatrix} \\cdot \\begin{bmatrix} b_1 \\\\ b_2 \\\\ \\ldots \\\\ b_n \\end{bmatrix} = a_{1}b_{1} + a_{2}b_{2} + \\ldots + a_{n}b_{n}$ 可以看出，向量點積得到的是一個標量；<br>\n",
    "例如: $\\begin{bmatrix} 3 \\\\ 5 \\\\ 2 \\end{bmatrix} \\cdot \\begin{bmatrix} 1 \\\\ 4 \\\\ 7 \\end{bmatrix} = 3 \\cdot 1 + 5 \\cdot 4 + 2 \\cdot 7 = 37$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "vec_1 = np.array([3,5,2])\n",
    "vec_2 = np.array([1,4,7])\n",
    "print(np.dot(vec_1, vec_2)) # dot product of vectors\n",
    "print(np.cross(vec_1, vec_2)) # cross multipication of vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "soa = np.array([[0, 0, -1, 2], [0, 0, -3, 6]])\n",
    "X, Y, U, V = zip(*soa)\n",
    "plt.figure()\n",
    "ax = plt.gca()\n",
    "ax.quiver(X, Y, U, V, angles='xy', scale_units='xy', scale=1)\n",
    "ax.set_xlim([-3, 3])\n",
    "ax.set_ylim([-3, 8])\n",
    "plt.draw()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看出，第一個答案是點乘出來的結果，在`python-numpy`中，我們用`np.dot(vector, vector)`來計算兩個或多個向量的點積。而`(vector * vector)`則是計算向量的叉乘結果，下面來介紹一下叉乘（向量積）：<br>\n",
    "它是向量和向量的乘積，不過結果是一個向量，它的集合意義是所得的向量和被乘向量所在的平面垂直，方向由右手定則規定，大小是兩個被乘向量張成的平行四邊形的面積。所以，向量積不滿足交換律。下面主要介紹在$\\mathbf{R}^2$和$\\mathbf{R}^3$的情況下的向量積：<br>\n",
    "$\\mathbf{R}^2$的向量積：<br>\n",
    "<center>$\\begin{bmatrix} a_1 \\\\ a_2\\end{bmatrix} \\times \\begin{bmatrix} b_1 \\\\ b_2 \\end{bmatrix} = \\begin{bmatrix} a_1 \\cdot b_2 - a_2 \\cdot b_1\\end{bmatrix}$</center><br>\n",
    "例如：<center>$\\begin{bmatrix} 1 \\\\ 2 \\end{bmatrix} \\times \\begin{bmatrix} 3 \\\\4 \\end{bmatrix} \n",
    "=\\begin{bmatrix} 1 \\cdot 4 - 3 \\cdot 2 \\end{bmatrix}= \\begin{bmatrix}-2\\end{bmatrix}$</center>\n",
    "\n",
    "$\\mathbf{R}^3$的向量積：<br>\n",
    "<center>$\\begin{bmatrix} a_1 \\\\ a_2 \\\\ a_3\\end{bmatrix} \\times \\begin{bmatrix} b_1 \\\\ b_2 \\\\ b_3 \\end{bmatrix} = \\begin{bmatrix} a_2 \\cdot b_3 - a_3 \\cdot b_2 \\\\ a_3 \\cdot b_1 - a_1 \\cdot b_3 \\\\ a_1 \\cdot b_2 - a_2 \\cdot b_1\\end{bmatrix}$</center><br>\n",
    "例如：<center>$\\begin{bmatrix} 3 \\\\ 5 \\\\ 2 \\end{bmatrix} \\times \\begin{bmatrix} 1 \\\\ 4 \\\\ 7 \\end{bmatrix} =\\begin{bmatrix} 5 \\cdot 7 - 2 \\cdot 4 \\\\ 2 \\cdot 1 - 3 \\cdot 7 \\\\ 3 \\cdot 4 - 5 \\cdot 1\\end{bmatrix}= \\begin{bmatrix} 27 \\\\ -19 \\\\ 7\\end{bmatrix}$</center><br>\n",
    "可以看出，向量積的結果就是一個新的向量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "a = np.array([3, 5, 2])\n",
    "b = np.array([1, 4, 7])\n",
    "print(np.cross(a, b)) # cross multipication of vec a and vec b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "之前我們所講的向量長度的定義式：$\\left\\| \\vec{\\mathbf{v}} \\right\\| = \\sqrt{v_{1}^{2} + v_{2}^{2} + \\ldots + v_{n}^{2}}$，如果我們連立向量點積的定義就可以得出$\\parallel \\mathbf{v} \\parallel = \\sqrt{\\vec{v}\\cdot \\vec{v}}$<br>\n",
    "關於點積，還有一個重要的重要的性質，就是**柯西不等式**：\n",
    "**對於兩個非零向量，$\\vec{x}, \\vec{y} \\in \\mathbf{R}^n, |\\vec{x}\\cdot \\vec{y}| \\leq \\parallel \\mathbf{x}\\parallel \\parallel\\mathbf{y} \\parallel$，當且僅當$\\vec{\\mathbf{x}} = c\\vec{\\mathbf{y}}$時，等號成立**。證明如下：<br>\n",
    "<center>$\\begin{align}\n",
    "\\left\\|\\vec{\\mathbf{x}} + \\vec{\\mathbf{y}}\\right\\|^2 & = (\\vec{\\mathbf{x}} + \\vec{\\mathbf{y}})\\cdot (\\vec{\\mathbf{x}} + \\vec{\\mathbf{y}}) \\\\\\\n",
    " & = \\left\\|\\vec{\\mathbf{x}}\\right\\|^2 + 2\\vec{\\mathbf{x}}\\vec{\\mathbf{y}} + \\left\\|\\vec{\\mathbf{y}}\\right\\|^2 \\\\\\\n",
    "& \\le \n",
    "\\left\\|\\vec{\\mathbf{x}}\\right\\|^2 + 2\\left\\|\\vec{\\mathbf{x}}\\right\\|\\left\\|\\vec{\\mathbf{y}}\\right\\| + \\left\\|\\vec{\\mathbf{y}}\\right\\|^2\n",
    "\\end{align}$</center><br>\n",
    "所以：<br>\n",
    "<center>$\\left\\|\\vec{\\mathbf{x}} + \\vec{\\mathbf{y}}\\right\\|^2 \\le (\\left\\|\\vec{\\mathbf{x}}\\right\\| + \\left\\|\\vec{\\mathbf{y}}\\right\\|)^2$</center><br>\n",
    "兩邊開平方後：<br>\n",
    "<center>$\\left\\|\\vec{\\mathbf{x}} + \\vec{\\mathbf{y}}\\right\\| \\le \n",
    "\\left\\|\\vec{\\mathbf{x}}\\right\\| + \\left\\|\\vec{\\mathbf{y}}\\right\\|$</center><br>\n",
    "這也就是我們高中三角函數裡學過的**三角不等式**，從幾何的角度來說，向量的點積和向量的夾角$\\theta$的餘弦值時有關的：$\\vec{\\mathbf{a}}\\cdot\\vec{\\mathbf{b}} = \\left\\|\\vec{\\mathbf{a}}\\right\\|\\left\\|\\vec{\\mathbf{b}}\\right\\|cos\\theta$                                                                      "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "從另一個角度來看，這其實就是一個向量在另一個向量上的投影: $Proj_L({\\vec{\\mathbf{x}}}) =  c\\vec{\\mathbf{v}} = (\\frac{\\vec{\\mathbf{x}}\\cdot \\vec{\\mathbf{v}}}{\\vec{\\mathbf{v}}\\cdot \\vec{\\mathbf{v}}})\\vec{\\mathbf{v}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "#the first way to define the function\n",
    "def get_projection(a, b):\n",
    "    return a.dot(b)*1.0*b/b.dot(b)\n",
    "\n",
    "a = np.array([1, 2])\n",
    "b = np.array([2, 2])\n",
    "print(get_projection(a, b))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "#the second way to define the function \n",
    "find_projection = lambda a, b : a.dot(b) * 1.0 * b/b.dot(b)\n",
    "\n",
    "a = np.array([1, 2])\n",
    "b = np.array([2, 2])\n",
    "\n",
    "print(find_projection(a,b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這時候再考慮向量積，從集合的角度，向量積和向量間的夾角$\\theta$的正弦值有關：$\\left\\|\\vec{\\mathbf{a}}\\times\\vec{\\mathbf{b}}\\right\\| = \\left\\|\\vec{\\mathbf{a}}\\right\\|\\left\\|\\vec{\\mathbf{b}}\\right\\|sin\\theta$，這就意味著，向量的向量積反應了兩個向量的正交程度（正交：向量積等於零），$sin\\theta = 0$ 正交程度最小，$sin\\theta = 1$，正交程度最大。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 矩陣的向量積\n",
    "\n",
    "*矩陣乘法*\n",
    "我們可以將向量理解成一個$1 \\times n$的矩陣，當矩陣$A$的列數與向量$\\vec x$的分量數目想等式，矩陣和向量的積有定義：<br>\n",
    "<center>$\\underset{m\\times n}{A}\\vec{\\mathbf{x}}=\\begin{bmatrix}a_{11} & a_{12} & \\ldots & a_{1n} \\\\ a_{21} & a_{22} & \\ldots & a_{2n} \\\\ \\ldots \\\\ a_{m1} & a_{m2} & \\ldots & a_{mn}\\end{bmatrix}\\begin{bmatrix}x_1 \\\\ x_2 \\\\ \\ldots \\\\ x_n \\end{bmatrix} = \\begin{bmatrix}a_{11}x_1 + a_{12}x_2 + \\ldots + a_{1n}x_n \\\\ a_{21}x_1 + a_{22}x_2 + \\ldots + a_{2n}x_n \\\\ \\ldots \\\\ a_{m1}x_1 + a_{m2}x_2 + \\ldots + a_{mn}x_n \\\\ \\end{bmatrix}$</center><br><br>\n",
    "例：$\\begin{bmatrix}4 & 3 & 1 \\\\ 1 & 2 & 5\\end{bmatrix}  \\begin{bmatrix}5 \\\\ 2 \\\\ 7\\end{bmatrix} = \\begin{bmatrix}4\\cdot 5 + 3\\cdot 2 + 1\\cdot 7 \\\\ 1 \\cdot 5 + 2 \\cdot 2 + 5 \\cdot 7\\end{bmatrix} = \\begin{bmatrix}33 \\\\ 44\\end{bmatrix}$</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "矩陣的向量積，可以當作時矩陣的所有列向量的線性組合：<br>\n",
    "$\\underset { m\\times n }{ \\mathbf{A} } \\vec { \\mathbf{x} } =\\begin{bmatrix} \\underbrace { \\begin{bmatrix} a_{ 11 } \\\\ a_{ 21 } \\\\ \\ldots \\\\ a_{ m1 } \\end{bmatrix} }_{ \\vec { \\mathbf{ V }_{ 1 } }  }  & \\underbrace { \\begin{bmatrix} a_{ 12 } \\\\ a_{ 22 } \\\\\\ldots  \\\\ a_{ m2 } \\end{bmatrix} }_{ \\vec { \\mathbf{ V_{ 2 } } }  } & \\ldots & \\underbrace { \\begin{bmatrix} a_{ 1n } \\\\ a_{ 2n } \\\\ \\ldots \\\\ a_{ mn } \\end{bmatrix} }_{ \\vec { \\mathbf{ V_{ n } } }  }  \\end{bmatrix}\\begin{bmatrix} x_{ 1 } \\\\ x_{ 2 } \\\\ \\ldots \\\\ x_{ n } \\end{bmatrix}=x_1\\vec{\\mathbf{V}_1}+x_2\\vec{\\mathbf{V}_2}+\\ldots+x_n\\vec{\\mathbf{V}_n}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "而向量$\\vec{x}$的每一個份量可以看成$\\mathbf{A}$的每一個列向量的加權。**一個矩陣其實就是一個線性變換，一個矩陣乘以一個向量後得到的向量，其實就相當於將這個向量進行了線性變換。**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 向量的轉置 \n",
    "\n",
    "向量$ \\vec{\\mathbf{V}} = \\underbrace{\\begin{bmatrix}v_1 \\\\ v_2 \\\\ \\ldots \\\\ v_n \\end{bmatrix}}_{n\\times 1}$ 的轉置定義為$\\mathbf{V}^T = \\underbrace{\\begin{bmatrix}v_1 & v_2 & \\ldots & v_n \\end{bmatrix}}_{1 \\times n}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we use V = (2,4) to give a example\n",
    "import numpy as np\n",
    "\n",
    "vec_3 = np.array([[2,4]]) # if we use double bracket, we are writing a two dimentional vecotr\n",
    "print(vec_3.T) # print the transport of vec_3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "向量的轉置有一個性質：一個向量$\\vec {v}$ 點乘另一個向量$\\vec{w}$，其結果和向量$\\vec{v}$轉置後和向量$\\vec{w}$做矩陣乘法相同。即：$\\vec{\\mathbf{v}} \\cdot \\vec{\\mathbf{w}} = \\vec{\\mathbf{v}}^T \\vec{\\mathbf{w}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chapter Four Vector Space\n",
    "\n",
    "### 張成空間（span）\n",
    "\n",
    "一組向量的張成空間，說白了就是指這些向量隨便線性組合後能夠表示多少個向量，也被稱為線性生成空間。記做$span(S)$；如果$S = \\{v_1, v_2, \\dots, v_n\\}$是 $V$的有限子集，則生成空間為：$span(S) = span(v_1, v_2, \\dots, v_n) = \\{\\lambda_1 v_1, \\lambda_2 v_2, \\dots, \\lambda_n v_n|\\lambda_1, \\lambda_2, \\dots, \\lambda_n \\in K\\}$\n",
    "<br>\n",
    "例如，對於$\\mathbf{R}^2$空間中兩個不平行的非零向量$\\vec{vec_4} = \\begin{bmatrix} 2 \\\\ 1 \\end{bmatrix}$和向量$\\vec{vec_5} = \\begin{bmatrix} 0 \\\\ 3 \\end{bmatrix}$，不難發現，這兩個向量可以表示二維空間$\\mathbf{R}^2$中的任意一個向量，即，$span(\\vec{vec_4}, \\vec{vec_5}) = \\mathbf{R}^2$，證明如下："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "對於$\\mathbf{R}^2$中任意向量$\\begin{bmatrix} x \\\\ y \\end{bmatrix}$，假設可以用$vec_4$和$vec_5$線性組合而成，<br>\n",
    "那麼有：$c_1 \\begin{bmatrix}2 \\\\ 1\\end{bmatrix} + c_2 \\begin{bmatrix} 0 \\\\ 3 \\end{bmatrix} = \\begin{bmatrix} x \\\\ y \\end{bmatrix}$<br>\n",
    "即：$\\left\\{\n",
    "\\begin{align}\n",
    "c_1 \\cdot 2 & + c_2 \\cdot 0 &= x\\\\\\\n",
    "c_1 \\cdot 1 & + c_2 \\cdot 3 &= y\n",
    "\\end{align}\n",
    "\\right.$ <br>\n",
    "求解得：\n",
    "$\\left\\{\n",
    "\\begin{align}\n",
    "c_1 &= \\frac{x}{2}\\\\\n",
    "c_2 &= \\frac{y}{3} - \\frac{x}{6}\n",
    "\\end{align}\n",
    "\\right.$<br>\n",
    "由於$x, y$的值已經確定，所以$c_1，c_2$的值也必然是唯一的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 線性相關和線性無關（Linearly Dependent and Linearly Independent）\n",
    "\n",
    "在線性代數裡，向量空間的一組元素中，若沒有向量可用有限個其他向量的線性組合所表示，则稱為線性無關或線性獨立（linearly independent），反之稱為線性相關（linearly dependent）。例如在三維歐幾里得空間$\\mathbf{R}^3$的三個向量$(1, 0, 0)，(0, 1, 0)$ 和$(0, 0, 1)$線性無關。但$(2, −1, 1)，(1, 0, 1)$和 $(3, −1, 2)$線性相關，因為第三個是前兩個的和。<br>\n",
    "*相信各位都已知道那八條還是十條相關性，這裡不做贅述*<br>\n",
    "上面那個例子例如，$\\vec{vec_4} = \\begin{bmatrix} 2 \\\\ 1 \\end{bmatrix}$和向量$\\vec{vec_5} = \\begin{bmatrix} 0 \\\\ 3 \\end{bmatrix}$，如果給他們再加一個向量$\\vec{vec_6} = \\begin{bmatrix} 5 \\\\ 2 \\end{bmatrix}$，由於$\\vec{vec_5}$可以由其他兩個向量線性組合而成，由三個向量共同張成的空間並沒有發生變化，仍然是$\\mathbf{R}^3$，因此稱集合$\\{\\vec{vec_3}, \\vec{vec_4}, \\vec{vec_5}\\}$線性相關。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 如何判斷是否線性相關\n",
    "\n",
    "一個向量集合$s = v_1, v_2, \\dots, v_n$ 線性相關的充分必要是存在一部分非零係數使得：$c_1v_1 + c_2v_2 + \\dots + c_nv_n = \\begin{bmatrix} 0 \\\\ 0 \\\\ \\dots \\\\ 0 \\end{bmatrix}$。<br>\n",
    "例如：向量：$\\begin{bmatrix}2 \\\\ 1\\end{bmatrix}$ 和 $\\begin{bmatrix}3 \\\\ 2\\end{bmatrix}$，則先寫出如下等式：\n",
    "$c_1 \\begin{bmatrix}2 \\\\ 1\\end{bmatrix} + c_2 \\begin{bmatrix}3 \\\\ 2\\end{bmatrix} = \\begin{bmatrix}0 \\\\ 0\\end{bmatrix}$,<br>\n",
    "容易求解得：$\\begin{bmatrix}c_1 \\\\ c_2\\end{bmatrix} = \\begin{bmatrix}0 \\\\ 0\\end{bmatrix}$，說明這兩個向量線性無關。<br>\n",
    "類似的，對於三個$\\mathbf{R}^3$中的向量$\\begin{bmatrix}2 \\\\ 0 \\\\ 0\\end{bmatrix}, \\begin{bmatrix} 0 \\\\ 1 \\\\ 0\\end{bmatrix}$和$\\begin{bmatrix} 0 \\\\ 0 \\\\ 7\\end{bmatrix}$，不難判斷出它們是線性無關的，它們共同張成了$\\mathbf{R}^3$空間。<br>\n",
    "而對於向量集合：$\\left\\{\\begin{bmatrix}2 \\\\ 1\\end{bmatrix}, \\begin{bmatrix}3 \\\\ 2\\end{bmatrix}, \\begin{bmatrix}1 \\\\ 2 \\end{bmatrix}\\right\\}$，不難算出存在非零係數：$\\begin{bmatrix}c_1 \\\\ c_2 \\\\ c_3\\end{bmatrix} = \\begin{bmatrix}-4 \\\\ 3 \\\\ -1\\end{bmatrix}$ 使得 $c1 \\begin{bmatrix}2 \\\\ 1\\end{bmatrix} + c_2 \\begin{bmatrix}3 \\\\ 2\\end{bmatrix} + c_3 \\begin{bmatrix}1 \\\\ 2 \\end{bmatrix} = \\begin{bmatrix}0 \\\\ 0\\end{bmatrix}$。因此，此集合線性相關。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chapter Five Linear Algebra Advanced Text\n",
    "\n",
    "在前面幾張，已經簡單介紹了線性代數中最基本的數據表達方式：**矩陣**和**向量**。有了這兩個數學工具作為基礎，我們可以進一步討論下面的內容：<br>\n",
    "1. 如何理解線性代數的基。<br>\n",
    "2. 向量中的子空間、另空間、列空間、行空間、零空間都是什麼？我們應該如何求解？<br>\n",
    "3. 如何用線性代數的知識來擬合數據？<br>\n",
    "4. 機器學習、圖形處理中常見的“特徵向量”究竟是什麼？它和變換矩陣有什麼關係？<br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 階梯形矩陣（Echelon Matrix）\n",
    "\n",
    "階梯形矩陣是一類非常實用的工具，可以幫助我們求解線性空間的基，這就是能用在諸如計算解不是唯一的方程組之類的問題上。\n",
    "<br>\n",
    "#### 階梯形矩陣\n",
    "\n",
    "若矩陣$\\mathbf{A}$滿足如下條件：<br>\n",
    "1. 若有零行（元素全為零的行），則零行應在最下方；<br>\n",
    "2. 非零首元（即非零行的第一個不為零的元素）的列標號隨行標號的增加而嚴格遞增。<br>\n",
    "\n",
    "則這個矩陣稱為階梯形矩陣。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "示例：<br>\n",
    "<center>$\\begin{bmatrix}\n",
    "2 & 0 & 2 & 1 \\\\\n",
    "0 & 5 & 2 & -2 \\\\\n",
    "0 & 0 & 3 & 2 \\\\\n",
    "0 & 0 & 0 & 0\n",
    "\\end{bmatrix}$</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 行簡化階梯形矩陣（Row Echelon Matrix）\n",
    "\n",
    "若矩陣$\\mathbf{A}$滿足如下條件：<br>\n",
    "1. 它是階梯形矩陣;<br>\n",
    "2. 非零首元所在列出了非零首元外，其餘元素都為零。<br>\n",
    "\n",
    "則稱此矩陣為行簡化階梯形矩陣。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "示例：<br>\n",
    "<center>$\\begin{bmatrix}\n",
    "2 & 0 & 2 & 1 \\\\\n",
    "0 & 5 & 2 & -2 \\\\\n",
    "0 & 0 & 3 & 2 \\\\\n",
    "0 & 0 & 0 & 0\n",
    "\\end{bmatrix}$</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 行最簡階梯形矩陣（reduced row echelon form）\n",
    "\n",
    "若矩陣$\\mathbf{A}$滿足如下條件：<br>\n",
    "1. 它是行簡化階梯形矩陣；<br>\n",
    "2. 非零首元都為一。<br>\n",
    "\n",
    "則稱它為行最簡階梯形矩陣。下面我們來講講如何將矩陣化簡為最簡階梯形：<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "對於這個矩陣$\\mathbf{A} = \\begin{bmatrix}\n",
    "1 & 2 & 1 & 1 & 7\\\\ \n",
    "1 & 2 & 2 & -1 & 12\\\\ \n",
    "2 & 4 & 0 & 6 & 4\n",
    "\\end{bmatrix}$，使用**初等變換（Elementary Operations of Linear System）**就可以將矩陣轉換成如下形式：\n",
    "$\\begin{bmatrix}\n",
    "1 & 2 & 1 & 1 & 7\\\\ \n",
    "1 & 2 & 2 & -1 & 12\\\\ \n",
    "2 & 4 & 0 & 6 & 4\n",
    "\\end{bmatrix}\n",
    "\\rightarrow\n",
    "\\begin{bmatrix}\n",
    "1 & 2 & 1 & 1 & 7\\\\ \n",
    "0 & 0 & 1 & -2 & 5\\\\ \n",
    "2 & 4 & 0 & 6 & 4\n",
    "\\end{bmatrix}\n",
    "\\rightarrow\n",
    "\\begin{bmatrix}\n",
    "1 & 2 & 1 & 1 & 7\\\\ \n",
    "0 & 0 & 1 & -2 & 5\\\\ \n",
    "0 & 0 & -2 & 4 & -10\n",
    "\\end{bmatrix}\n",
    "\\rightarrow\n",
    "\\begin{bmatrix}\n",
    "1 & 2 & 1 & 1 & 7\\\\ \n",
    "0 & 0 & 1 & -2 & 5\\\\ \n",
    "0 & 0 & 0 & 0 & 0\n",
    "\\end{bmatrix}\n",
    "\\rightarrow\n",
    "\\begin{bmatrix}\n",
    "1 & 2 & 0 & 3 & 2\\\\ \n",
    "0 & 0 & 1 & -2 & 5\\\\ \n",
    "0 & 0 & 0 & 0 & 0\n",
    "\\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**行最簡形**非常實用，可以用來解方程（滑稽；例如下面這個方程組：<br>\n",
    "<center>$\\left\\{ \n",
    "\\begin{eqnarray} \n",
    "x_1 + 2x_2 + x_3 + x_4 &=& 7 \\\\\\\n",
    "x_1 + 2x_2 + 2x_3 - x_4 &=& 12 \\\\\\\n",
    "2x_1 + 4x_2 + 6x_4 &=& 4\n",
    "\\end{eqnarray}\n",
    "\\right.$</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "按照常理，只有三個方程，肯定無法求解出四個未知數，此時，如果我們使用`numpy.linalg.solve`，求解時就會觸發`LinAlgError`，但是通過化成最簡形，我們可以進一步找出變量的憲制關係。首先，將方程組表達成增廣矩陣形式：<br>\n",
    "<center>$\\begin{bmatrix}\n",
    "1 & 2 & 1 & 1 & 7\\\\ \n",
    "1 & 2 & 2 & -1 & 12\\\\ \n",
    "2 & 4 & 0 & 6 & 4\n",
    "\\end{bmatrix}$</center>\n",
    "<br>\n",
    "然後將其行簡化，得：<br>\n",
    "<center>$\\begin{bmatrix}\n",
    "1 & 2 & 0 & 3 & 2\\\\ \n",
    "0 & 0 & 1 & -2 & 5\\\\ \n",
    "0 & 0 & 0 & 0 & 0\n",
    "\\end{bmatrix}$</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "從結果可以看出這個矩陣的主列有兩個，而且是線性無關的。所以矩陣$A$的秩是$2$，，即，$rank\\mathbf{(A)} = 2$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**矩陣的秩的一個特性：矩陣$A$等於矩陣$A$的轉置地秩序，即，**$rank(\\mathbf{A})=rank(\\mathbf{A^T})$<br>\n",
    "在`python`中，可以使用`numpy`包中的`linalg.matrix_rank`方法計算矩陣的秩："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "a = np.matrix('1 1 1 1;1 2 3 4;4 3 2 1')\n",
    "print(np.linalg.matrix_rank(a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**注意**⚠️，在`numpy`中的秩和線性代數裡的秩不是同一個概念。在`numpy`中維度（dimensions）叫做軸（axes），軸的個數叫做秩。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "a = np.matrix('1 1 1 1;1 2 3 4; 0 0 1 0')\n",
    "print (a.ndim)  # 2（维度）\n",
    "print (np.rank(a))  # 2（a.ndim 的别名，已经过时）\n",
    "print (np.linalg.matrix_rank(a))  # 3（秩）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 行空間\n",
    "\n",
    "有了列空間的定義，行空間顧名思義其實就是矩陣每一行轉置得到的向量張成的自空間，也就是矩陣的轉置的列空間，記做：$R(\\mathbf{A}) = C(\\mathbf{A}^T)$<br>\n",
    "例如：矩陣$\\mathbf{A} = \\begin{bmatrix}1 & 1 & 1 & 1 \\\\ 1 & 2 & 3 & 4 \\\\4 & 3 & 2 & 1\\end{bmatrix}$的行空間就等於：$R(\\mathbf{A}) = C(\\mathbf{A}^T) = span\\left(\\begin{bmatrix}1 \\\\ 1 \\\\ 1 \\\\ 1\\end{bmatrix}\\begin{bmatrix}1 \\\\ 2 \\\\ 3 \\\\ 4\\end{bmatrix}\\begin{bmatrix}4 \\\\ 3 \\\\ 2 \\\\ 1\\end{bmatrix}\\right)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 零空間\n",
    "\n",
    "上面已經講過了零空間，現在這裡的零空間是指矩陣$A^T$的零空間，也就是**左零空間**。即：$N(\\mathbf{A}^T) = \\left\\{ \\vec{\\mathbf{x}} | \\mathbf{A}^{T} \\vec{\\mathbf{x}} = \\vec{\\mathbf{0}} \\right\\} = \\left\\{ \\vec{\\mathbf{x}} | \\vec{\\mathbf{x}}^{T} \\mathbf{A} = \\vec{\\mathbf{0}}^{T} \\right\\}$<br>\n",
    "例如，矩陣$\\mathbf{B} = \\begin{bmatrix}1 & 1 & 4 \\\\ 1 & 2 & 3 \\\\1 & 4 & 2\\\\ 1 & 3 & 1\\end{bmatrix}$的轉置是矩陣$\\mathbf{A} = \\mathbf{A} = \\begin{bmatrix}1 & 1 & 1 & 1 \\\\ 1 & 2 & 3 & 4 \\\\4 & 3 & 2 & 1\\end{bmatrix}$，因此左零空間等於：$N(\\mathbf{B^T}) = N(\\mathbf{A}) = span\\left(\\begin{bmatrix} 1 \\\\ -2 \\\\ 1 \\\\ 0 \\end{bmatrix} \\begin{bmatrix} 2 \\\\ -3 \\\\ 0 \\\\ 1 \\end{bmatrix}\\right)$；由於轉置是對稱的，所以矩陣$A$的轉置的左零空間也是矩陣$A$的零空間。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 子空間的正交補\n",
    "\n",
    "假設$V$是$R^n$的一個子空間，那麼$V$的正交補$V^\\perp$也是一個子空間，定義為$\\left\\{\\vec{\\mathbf{x}} | \\vec{\\mathbf{x}} \\vec{\\mathbf{v}}=0\\right\\}$，也就是說$R^n$中所有正交於$V$的向量所組成的子空間。<br>\n",
    "由於正交是對稱的，所以正交補也是對稱的。一個子空間的正交補的正交補依然等於這個子空間。<br>\n",
    "矩陣的零空間是行空間的正交補：$N(\\mathbf{A}) = R(\\mathbf{A})^{\\bot}$；反過來，矩陣的左零空間是列空間的正交補：$N(\\mathbf{B}^T) = C(\\mathbf{B})^{\\bot}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 最小二乘逼近\n",
    "\n",
    "最小二乘法是一個使用的數學工具，利用它在方程誤解的情況下給出近似解。在機器學習中，最小二乘逼近是一個重要的擬合方法。<br>\n",
    "假設有一個方程：$\\underset{n\\times k}{\\mathbf{A}}\\vec{\\mathbf{x}} = \\vec{\\mathbf{b}}$<br>\n",
    "無解。把上式寫成：$\\vec{a_1}\\vec{\\mathbf{x}} + \\vec{a_2}\\vec{\\mathbf{x}} + \\ldots + \\vec{a_k}\\vec{\\mathbf{x}} = \\vec{\\mathbf{b}}$<br>\n",
    "無解，就意味著$\\mathbf{A}$的所有列向量的張成空間不包括向量$\\vec{b}$。即：$\\vec{\\mathbf{b}} \\notin span(C(\\mathbf{A}))$<br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們可以通過最小二乘法，求解出近似解。即，要找出一些$\\vec{x*}$使得$\\left\\|\\vec{\\mathbf{b}}-\\mathbf{A}\\vec{\\mathbf{x}^*}\\right\\|$最小。用向量$\\vec{\\mathbf{V}}$代表$\\mathbf{A}\\vec{\\mathbf{x*}}$，有：<br>\n",
    "$\\left\\|\n",
    "\\begin{bmatrix}\n",
    "\\vec{b_1}-\\vec{v_1}\\\\\n",
    "\\vec{b_2}-\\vec{v_2}\\\\\n",
    "\\ldots\\\\\n",
    "\\vec{b_n}-\\vec{v_n}\\\\\n",
    "\\end{bmatrix}\n",
    "\\right\\|^2\n",
    "= \n",
    "(b_1-v_1)^2 + (b_2-v_2)^2 + \\ldots + (b_n-v_n)^2$<br>\n",
    "把這個值最小化的過程叫做**最小二乘逼近**。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如何求出$\\mathbf{A}\\vec{\\mathbf{x*}}$這個近似值呢？從集合上考慮，列空間可以看成空間中張成的一個平面，而平面$\\vec{b}$並不落在這個平面上。但我們知道，在這個平面上與向量$\\vec{b}$最接近的向量就是它的投影！所以，$\\mathbf{A}\\vec{\\mathbf{x}^*} = Proj_{C(\\mathbf{A})}\\vec{\\mathbf{b}}$，直接計算$Proj_{C(\\mathbf{A})}\\vec{\\mathbf{b}}$並不簡單。不過，$\\vec{\\mathbf{b}}-\\mathbf{A}\\vec{\\mathbf{x}}$其實就是$\\mathbf{A \\vec{x}}$的正交補，所以一個簡單的求解方法是將原來無解的方程做成一個$\\mathbf{A}$的轉置再求解：$\\mathbf{A}^T\\mathbf{A}\\vec{\\mathbf{x}^*} = \\mathbf{A}^T\\vec{\\mathbf{b}}$得出的解就是原方程的近似解。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 示例 解下列方程組\n",
    "$\\left\\{ \n",
    "\\begin{eqnarray} \n",
    "x + y &=& 3 \\\\\\\n",
    "x - y &=& -2 \\\\\\\n",
    "y &=& 1\n",
    "\\end{eqnarray}\n",
    "\\right.$<br>\n",
    "將三個方程表示的直線畫出來，可以看出這三條直線並沒有交點："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from pylab import *\n",
    "\n",
    "plt.xlim(-3,3)\n",
    "plt.ylim(0,6)\n",
    "plt.plot([-2,3],[0,5],'blue',[-3,3],[6,0],'yellow',[-3,3],[1,1],'red')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "那麼，我們應該如何找出一個與三條直線距離最近的一個點呢？這時候就可以使用最小二乘逼近了。<br>\n",
    "先將方程寫成矩陣和向量的形式：<br>\n",
    "<center>$\\underbrace{\n",
    "\\begin{bmatrix}\n",
    "1 & 1 \\\\\n",
    "1 & -1 \\\\\n",
    "0 & 1\n",
    "\\end{bmatrix}\n",
    "}_{\\mathbf{A}}\n",
    "\\underbrace{\n",
    "\\begin{bmatrix}\n",
    "x \\\\\n",
    "y\n",
    "\\end{bmatrix}\n",
    "}_{\\vec{\\mathbf{x}}}\n",
    "=\n",
    "\\underbrace{\n",
    "\\begin{bmatrix}\n",
    "3 \\\\\n",
    "-2 \\\\\n",
    "1\n",
    "\\end{bmatrix}\n",
    "}_{\\vec{\\mathbf{b}}}$</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這個等式的最小二乘逼近就是：<br>\n",
    "<center>$\\begin{align}\n",
    "\\begin{bmatrix}\n",
    "1 & 1 & 0 \\\\\n",
    "1 & -1 & 1\\\\\n",
    "\\end{bmatrix}\n",
    "\\begin{bmatrix}\n",
    "1 & 1 \\\\\n",
    "1 & -1 \\\\\n",
    "0 & 1\n",
    "\\end{bmatrix}\n",
    "\\begin{bmatrix}\n",
    "x^* \\\\\n",
    "y^*\n",
    "\\end{bmatrix}\n",
    "& = \n",
    "\\begin{bmatrix}\n",
    "1 & 1 & 0 \\\\\n",
    "1 & -1 & 1\\\\\n",
    "\\end{bmatrix}\n",
    "\\begin{bmatrix}\n",
    "3 \\\\\n",
    "-2 \\\\\n",
    "1\n",
    "\\end{bmatrix}\n",
    "\\\\\\\n",
    "\\begin{bmatrix}\n",
    "2 & 0 \\\\\n",
    "0 & 3\n",
    "\\end{bmatrix}\n",
    "\\begin{bmatrix}\n",
    "x^* \\\\\n",
    "y^*\n",
    "\\end{bmatrix}\n",
    "& =\n",
    "\\begin{bmatrix}\n",
    "1 \\\\\n",
    "6\n",
    "\\end{bmatrix}\n",
    "\\end{align}$</center>\n",
    "<br>\n",
    "\n",
    "由於是二解方程，我們可以很容易求出矩陣$\\begin{bmatrix}2 & 0 \\\\ 0 & 3\\end{bmatrix}$的逆是：$\\begin{bmatrix}\\frac{1}{2} & 0 \\\\ 0 & \\frac{1}{3}\\end{bmatrix}$<br>\n",
    "\n",
    "所以：\n",
    "<br>\n",
    "<center>\n",
    "$\\begin{bmatrix}\n",
    "x^* \\\\\n",
    "y^*\n",
    "\\end{bmatrix}\n",
    "=\n",
    "\\begin{bmatrix}\\frac{1}{2} & 0 \\\\ 0 & \\frac{1}{3}\\end{bmatrix}\n",
    "\\begin{bmatrix}\n",
    "1 \\\\\n",
    "6\n",
    "\\end{bmatrix}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "\\frac{1}{2} \\\\\n",
    "2\n",
    "\\end{bmatrix}$\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在`python`中，可以使用`numpy.linalg.lstsq`方法來求解最小二乘逼近。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from pylab import *\n",
    "\n",
    "x = [0.5]\n",
    "y = [2]\n",
    "plt.xlim(-3,3)\n",
    "plt.ylim(0,6)\n",
    "plt.plot([-2,3],[0,5],'blue',[-3,3],[6,0],'yellow',[-3,3],[1,1],'red',xData,yData1,'black')\n",
    "plt.scatter(x,y,color = 'black')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([[1, 1], [1, -1], [0, 1]])\n",
    "b = np.array([3, -2, 1])\n",
    "x = np.linalg.lstsq(a,b)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`numpy.inalg.lstsq`的返回值包括了四個部分：<br>\n",
    "1. 最小二乘逼近，如果`b`是二維的，那麼這個逼近的結果有多個列，每一列是一個逼近解，逼近解就是$\\begin{bmatrix}0,5 \\\\ 2 \\end{bmatrix}$.<br>\n",
    "2. 殘差。即，每一個`b - a*x`的長度的和。對於上例，殘差是$1.5$.<br>\n",
    "3. 矩陣$a$的秩。對於上例，矩陣$a$的秩為$2$。<br>\n",
    "4. 矩陣$a$的歧異值。對於上例，矩陣$a$的奇異值為：$\\begin{bmatrix}1.73205081 \\\\  1.41421356\\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 线性回归\n",
    "\n",
    "問題：給定$4$個坐標點$(-1,0),(0,1),(1,),(2,1)$，求一條經過這些點的直線$y = mx + b$。<br>\n",
    "將四個點畫圖如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from pylab import *\n",
    "\n",
    "\n",
    "x = [-1,0,1,2]\n",
    "y =[0,1,2,1]\n",
    "\n",
    "plt.scatter(x,y,color = 'black')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "顯然這樣的直線並不存在。然而我們能夠使用最小二乘逼近，找到一條盡可能靠近這些點的直線。將四個點表示成方程組的形式：<br>\n",
    "<center>\n",
    "$\\left\\{\n",
    "\\begin{eqnarray}\n",
    "f(-1) &= -m + b = 0\\\\\\\n",
    "f(0) &= 0 + b  = 1\\\\\\\n",
    "f(1) &= m + b = 2\\\\\\\n",
    "f(2) &= 2m + b = 1\n",
    "\\end{eqnarray}\n",
    "\\right.$\n",
    "</center>\n",
    "然後方程組用矩陣和向量的形式：<br>\n",
    "<center>\n",
    "    $\\underbrace{\n",
    "\\begin{bmatrix}\n",
    "-1 & 1 \\\\\n",
    "0 & 1 \\\\\n",
    "1 & 1 \\\\\n",
    "2 & 1\n",
    "\\end{bmatrix}\n",
    "}_{\\mathbf{A}}\n",
    "\\underbrace{\n",
    "\\begin{bmatrix}\n",
    "m\\\\\n",
    "b\n",
    "\\end{bmatrix}\n",
    "}_{\\vec{\\mathbf{x}}}\n",
    "=\n",
    "\\underbrace{\n",
    "\\begin{bmatrix}\n",
    "0\\\\\n",
    "1\\\\\n",
    "2\\\\\n",
    "1\n",
    "\\end{bmatrix}\n",
    "}_{\\vec{\\mathbf{b}}}$\n",
    "</center>\n",
    "這兩個等式的最小二乘逼近就是：<br><center>\n",
    "    $\\begin{align}\n",
    "\\begin{bmatrix}\n",
    "-1 & 0 & 1 & 2 \\\\\n",
    "1 & 1 & 1 & 1\n",
    "\\end{bmatrix}\n",
    "\\begin{bmatrix}\n",
    "-1 & 1 \\\\\n",
    "0 & 1 \\\\\n",
    "1 & 1 \\\\\n",
    "2 & 1\n",
    "\\end{bmatrix}\n",
    "\\begin{bmatrix}\n",
    "m^*\\\\\n",
    "b^*\n",
    "\\end{bmatrix}\n",
    "&=\n",
    "\\begin{bmatrix}\n",
    "-1 & 0 & 1 & 2 \\\\\n",
    "1 & 1 & 1 & 1\n",
    "\\end{bmatrix}\n",
    "\\begin{bmatrix}\n",
    "0\\\\\n",
    "1\\\\\n",
    "2\\\\\n",
    "1\n",
    "\\end{bmatrix}\\\\\\\n",
    "\\begin{bmatrix}\n",
    "6 & 2 \\\\\n",
    "2 & 4\n",
    "\\end{bmatrix}\n",
    "\\begin{bmatrix}\n",
    "m^*\\\\\n",
    "b^*\n",
    "\\end{bmatrix}\n",
    "&=\n",
    "\\begin{bmatrix}\n",
    "4\\\\\n",
    "4\n",
    "\\end{bmatrix}\n",
    "\\end{align}$</center>\n",
    "\n",
    "容易求得$\\begin{bmatrix}6 & 2\\\\2 & 4\\end{bmatrix}$的逆$\\frac{1}{20}\\begin{bmatrix}4 & -2\\\\-2 & 6\\end{bmatrix}$，因此<br><center>\n",
    "    $\\begin{bmatrix}m^*\\\\b^*\\end{bmatrix} = \\frac{1}{20}\\begin{bmatrix}4 & -2\\\\-2 & 6\\end{bmatrix}\\begin{bmatrix}4 \\\\ 4\\end{bmatrix} = \\frac{1}{20}\\begin{bmatrix}8 \\\\ 16\\end{bmatrix} = \\begin{bmatrix}\\frac{2}{5} \\\\ \\frac{4}{5}\\end{bmatrix}$\n",
    "</center>\n",
    "    \n",
    "將直線$y = \\frac{2}{5}x + \\frac{4}{5}$繪圖如下所示：    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from pylab import *\n",
    "\n",
    "\n",
    "x = [-1,0,1,2]\n",
    "y =[0,1,2,1]\n",
    "x1 = np.linspace(-3,3)\n",
    "y1 = lambda x: 2*x/5 + 4/5\n",
    "plt.scatter(x,y,color = 'black')\n",
    "plt.plot(x1,y1(x1),'r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這就是所求的直線的近似解。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.matrix('-1 1;0 1;1 1;2 1')\n",
    "b = np.array([0, 1, 2, 1])\n",
    "x = np.linalg.lstsq(a, b)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 特徵向量（Elgen Vector）\n",
    "\n",
    "“特徵”在模式識別和圖像處理中是一個非常常見的詞彙，我們要認識和描繪一件事物，首先就要找出這個事務的特徵。同樣的道理，要讓計算機識別一件事物，稍嫌就要讓計算機學會理解或者抽象出食物的特徵。什麼樣的東西能當成特徵呢？那必須是能“放之四海而皆準”的依據。不論個體如何變幻，都能從中吵到這類群裡共有的特點。<br>\n",
    "在線性代數中，“特徵”就是一個更抽象的描述。我們知道，矩陣乘法對應了一個變換，是把任意一個向量變換成裡一個方向或長度都大多不同的新向量。在這個變幻的過程中，原向量主要發成旋轉、伸縮的變化。如果矩陣對某一個向量或某些向量只發生伸縮（尺度）變換，而沒有產生旋轉的效果（也就意味著張成的子空間發生改變），這樣的向量就被認為是特徵向量。<br>\n",
    "<center>\n",
    "    $\\mathbf{T}(\\vec{\\mathbf{v}}) = \\underbrace{\\mathbf{A}}_{n\\times n}\\vec{\\mathbf{v}} = \\underbrace{\\lambda}_{特征值} \\overbrace{\\vec{\\mathbf{v}}}^{特征向量}$\n",
    "</center>\n",
    "\n",
    "其中，$\\mathbf{T}$是一種線性變換，我們知道線性變換，我們知道線性變換可以用矩陣向量積來表示，因此可以表示成$\\mathbf{A \\vec{v}}$。$\\mathbf{A}$是一個$n \\times n$的方陣。$\\vec{v}$就是特徵向量（Elgen Vector），也就是「能被伸縮的向量」（要求是非$0$向量），而$\\lambda$的特徵向量$\\vec{v}$所對應的特徵之，也就是「伸縮了多少」。如果特徵之是負數，那說明了矩陣不但吧向量拉長（縮短）了，而且讓向量只想了相反的方向。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**簡而言之，特徵就是在線性變量當中不變的向量**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 求解特徵值\n",
    "\n",
    "非$0$向量$\\vec{v}$是線性變化矩陣$\\mathbf{A}$的特徵向量，需要滿足如下條件：<br>\n",
    "\n",
    "<center>\n",
    "    $det(\\lambda \\mathbf{I}_n - \\underbrace{\\mathbf{A}}_{n\\times n}) = 0$\n",
    "</center>\n",
    "\n",
    "其中，$det$表示矩陣行列式，$\\lambda$是特徵值，$\\mathbf{I}$是單位矩陣。<br>\n",
    "例如矩陣$\\mathbf{A} = \\begin{bmatrix}1 & 2 \\\\ 4 & 3\\end{bmatrix}$，代入公式$2$得：<br>\n",
    "<center>\n",
    "    $\\begin{align} det\\left( \\lambda \\begin{bmatrix} 1 & 0 \\\\ 0 & 1 \\end{bmatrix}-\\begin{bmatrix} 1 & 2 \\\\ 4 & 3 \\end{bmatrix} \\right)  &=0 \\\\ det\\left( \\begin{bmatrix} \\lambda  & 0 \\\\ 0 & \\lambda  \\end{bmatrix}-\\begin{bmatrix} 1 & 2 \\\\ 4 & 3 \\end{bmatrix} \\right)  &=0 \\\\ det\\left( \\begin{bmatrix} \\lambda -1 & -2 \\\\ -4 & \\lambda -3 \\end{bmatrix} \\right)  &=0 \\end{align}$\n",
    "</center>\n",
    "所以有：<br>\n",
    "<center>\n",
    "    $\\begin{align} (\\lambda -1)(\\lambda -3)-8 & =0 \\\\ \\lambda ^{ 2 }-4\\lambda -5 &=0 \\\\ (\\lambda - 5)(\\lambda +1) &= 0\\end{align}$\n",
    "</center>\n",
    "因此$\\lambda$得值等於$5$或者$-1$。<br>\n",
    "在`python`中，可以使用`numpy.linalg.eigvals`方法求解一個方陣的特徵值："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.matrix('1 2;4 3')\n",
    "print(np.linalg.eigvals(a))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "b = np.matrix('1 2 3;4 3 1')#如果矩陣不是方針，就會出現 LinalgError 錯誤\n",
    "print(np.linalg.eigvals(b) )  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 求解特徵向量\n",
    "\n",
    "變換矩陣$\\mathbf{A}$的特徵空間（特徵向量的空間）可以用下面的等式來求解： <br>\n",
    "<center>\n",
    "    $\\mathbf{E}_{\\lambda}=N(\\lambda I_n - \\mathbf{A})$\n",
    "</center>\n",
    "\n",
    "例如上面的變換矩陣$\\mathbf{A} = \\begin{bmatrix}1 & 2 \\\\ 4 & 3 \\end{bmatrix}$，帶入公式$3$得：<br>\n",
    "<center>\n",
    "    ${ E }_{ \\lambda  }=N\\left( \\lambda I_{ n }-\\begin{bmatrix} 1 & 2 \\\\ 4 & 3 \\end{bmatrix} \\right) =N\\left( \\lambda \\begin{bmatrix} 1 & 0 \\\\ 0 & 1 \\end{bmatrix}-\\begin{bmatrix} 1 & 2 \\\\ 4 & 3 \\end{bmatrix} \\right) =N\\left( \\begin{bmatrix} \\lambda -1 & -2 \\\\ -4 & \\lambda -3 \\end{bmatrix} \\right)$\n",
    "</center>\n",
    "當$\\lambda = 5$時， <br>\n",
    "<center>\n",
    "    ${ E }_{ 5  }=N\\left( \\begin{bmatrix} 4 & -2 \\\\ -4 & 2 \\end{bmatrix} \\right)$\n",
    "</center>\n",
    "利用前面所學的零空間的求解方法，得：<br>\n",
    "<center>\n",
    "    ${ E }_{ 5  }= span\\left(\\begin{bmatrix}\\frac{1}{2} \\\\ 1 \\end{bmatrix}\\right)$\n",
    "</center>\n",
    "同樣的，當$\\lambda = -1$時，<br>\n",
    "<center>\n",
    "    ${ E }_{ -1  }= span\\left(\\begin{bmatrix}1 \\\\ -1 \\end{bmatrix}\\right)$\n",
    "</center> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在`python`中，可以使用`numpy.linalg.eig`方法來求解方針的特徵值和特徵向量："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.matrix('1 2;4 3')\n",
    "print(np.linalg.eig(a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "得到的元祖中，第一部分時特徵值，和前面使用`numpy.linalg.eigvals`得到的結果完全一樣；第二部分時特徵向量，乍一看和我們上面求解的結果不一樣，但如果我們這麼些就完全不一樣了：<br>\n",
    "<center>\n",
    "    $\\begin{bmatrix}-0.70710678\\begin{bmatrix}1 \\\\ -1\\end{bmatrix} & -0.89442719\\begin{bmatrix}\\frac{1}{2} \\\\ 1\\end{bmatrix} \\end{bmatrix}$\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "變換矩陣線性無關的特徵向量特別適合作為空間的基，因為在這些方向上變換矩陣可以拉伸向量而不必扭曲和旋轉它，使得計算大為簡單。我們把這種基稱為**特徵基**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
